2026-10-29 20:20:37,452 [ 92520 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2026-10-29 20:20:37,452 [ 92520 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2026-10-29 20:20:37,452 [ 92520 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2026-10-29 20:20:37,452 [ 92520 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_t2xrta --privileged --dns-search='.' --memory=30703247360 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel0_0.jsonl --report-log-exclude-logs-on-passed-tests test_allow_feature_tier/test.py::test_allow_feature_tier_in_general_settings test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings_with_old_compatibility test_allow_feature_tier/test.py::test_allow_feature_tier_in_user test_allow_feature_tier/test.py::test_it_is_possible_to_enable_experimental_settings_in_default_profile test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active test_attach_table_normalizer/test.py::test_attach_substr test_attach_table_normalizer/test.py::test_attach_substr_restart 'test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[0]' 'test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[1048576]' test_backup_restore_new/test_cancel_backup.py::test_cancel_backup test_backup_restore_new/test_cancel_backup.py::test_shutdown_cancel_backup test_backup_restore_on_cluster/test_different_versions.py::test_different_versions test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async 'test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3]' 'test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3_other_bucket]' test_backup_restore_s3/test.py::test_backup_restore_system_tables_with_plain_rewritable_disk 'test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_local]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3_plain]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_local]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_s3_plain]' test_backup_restore_s3/test.py::test_backup_to_s3 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[auto_single]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_multipart]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_single]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_multipart]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_single]' test_backup_restore_s3/test.py::test_backup_to_s3_multipart test_backup_restore_s3/test.py::test_backup_to_s3_named_collection 'test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3]' 'test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_other_bucket]' 'test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_plain_rewritable]' test_backup_restore_s3/test.py::test_backup_to_s3_native_copy_multipart test_backup_restore_s3/test.py::test_backup_to_tar test_backup_restore_s3/test.py::test_backup_to_tar_bz2 test_backup_restore_s3/test.py::test_backup_to_tar_gz test_backup_restore_s3/test.py::test_backup_to_tar_lzma test_backup_restore_s3/test.py::test_backup_to_tar_xz test_backup_restore_s3/test.py::test_backup_to_tar_zst test_backup_restore_s3/test.py::test_backup_to_zip 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[False-True-False]' 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-False-False]' 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-False]' 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-True]' test_backup_restore_s3/test.py::test_incremental_backup_append_table_def test_backup_restore_s3/test.py::test_user_specific_auth test_create_query_constraints/test.py::test_create_query_const_constraints test_create_query_constraints/test.py::test_create_query_minmax_constraints test_database_delta/test.py::test_complex_table_schema test_database_delta/test.py::test_embedded_database_and_tables test_database_delta/test.py::test_multiple_schemes_tables test_ddl_on_cluster_stop_waiting_for_offline_hosts/test.py::test_stop_waiting_for_offline_hosts test_deduplicated_attached_part_rename/test.py::test_deduplicated_attached_part_renamed_after_attach 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[cache]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[direct]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[flat]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[hashed]' 'test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node0]' 'test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node1]' 'test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node0]' 'test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node1]' 'test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node0]' 'test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node1]' 'test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node0]' 'test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node1]' test_dictionaries_dependency/test.py::test_no_lazy_load test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_system_reload test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_timer test_dictionaries_update_and_reload/test.py::test_reload_after_fail_in_cache_dictionary test_dictionaries_update_and_reload/test.py::test_reload_after_loading test_dictionaries_update_and_reload/test.py::test_reload_while_loading test_dictionary_ddl_on_cluster/test.py::test_dictionary_ddl_on_cluster test_disk_access_storage/test.py::test_alter test_disk_access_storage/test.py::test_create test_disk_access_storage/test.py::test_drop test_disks_app_other_disk_types/test.py::test_disks_app_plain_rewritable test_distributed_config/test.py::test_distibuted_settings test_distributed_ddl_on_cross_replication/test.py::test_alter_ddl test_distributed_ddl_on_cross_replication/test.py::test_atomic_database test_distributed_ddl_on_cross_replication/test.py::test_non_query_with_table_ddl test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority]' 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative]' test_distributed_load_balancing/test.py::test_load_balancing_round_robin test_enable_user_name_access_type/test.py::test_startup_scripts test_encrypted_disk/test.py::test_add_keys test_encrypted_disk/test.py::test_add_keys_with_id 'test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False]' 'test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True]' 'test_encrypted_disk/test.py::test_backup_restore[File-local_policy-encrypted_policy-False]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_distributed_ddl_on_cross_replication/test.py::test_alter_ddl test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node0] test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] test_allow_feature_tier/test.py::test_allow_feature_tier_in_general_settings test_disk_access_storage/test.py::test_alter test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_system_reload test_encrypted_disk/test.py::test_add_keys test_database_delta/test.py::test_complex_table_schema [gw4] [ 1%] PASSED test_allow_feature_tier/test.py::test_allow_feature_tier_in_general_settings test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings [gw5] [ 2%] PASSED test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_system_reload test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_timer [gw8] [ 3%] PASSED test_encrypted_disk/test.py::test_add_keys test_encrypted_disk/test.py::test_add_keys_with_id [gw9] [ 4%] PASSED test_distributed_ddl_on_cross_replication/test.py::test_alter_ddl test_distributed_ddl_on_cross_replication/test.py::test_atomic_database [gw8] [ 5%] PASSED test_encrypted_disk/test.py::test_add_keys_with_id test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] [gw9] [ 6%] PASSED test_distributed_ddl_on_cross_replication/test.py::test_atomic_database test_distributed_ddl_on_cross_replication/test.py::test_non_query_with_table_ddl [gw4] [ 7%] PASSED test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings_with_old_compatibility [gw2] [ 8%] PASSED test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default [gw9] [ 9%] PASSED test_distributed_ddl_on_cross_replication/test.py::test_non_query_with_table_ddl [gw8] [ 10%] PASSED test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] [gw4] [ 11%] PASSED test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings_with_old_compatibility test_allow_feature_tier/test.py::test_allow_feature_tier_in_user [gw8] [ 12%] PASSED test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] test_encrypted_disk/test.py::test_backup_restore[File-local_policy-encrypted_policy-False] [gw0] [ 13%] PASSED test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3] test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3_other_bucket] [gw1] [ 14%] PASSED test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node0] [gw7] [ 15%] PASSED test_disk_access_storage/test.py::test_alter test_disk_access_storage/test.py::test_create [gw8] [ 16%] PASSED test_encrypted_disk/test.py::test_backup_restore[File-local_policy-encrypted_policy-False] [gw5] [ 17%] PASSED test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_timer test_dictionaries_update_and_reload/test.py::test_reload_after_fail_in_cache_dictionary test_attach_table_normalizer/test.py::test_attach_substr [gw0] [ 18%] PASSED test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3_other_bucket] test_backup_restore_s3/test.py::test_backup_restore_system_tables_with_plain_rewritable_disk [gw4] [ 19%] PASSED test_allow_feature_tier/test.py::test_allow_feature_tier_in_user test_allow_feature_tier/test.py::test_it_is_possible_to_enable_experimental_settings_in_default_profile test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node1] [gw4] [ 20%] PASSED test_allow_feature_tier/test.py::test_it_is_possible_to_enable_experimental_settings_in_default_profile test_backup_restore_new/test_cancel_backup.py::test_cancel_backup [gw7] [ 21%] PASSED test_disk_access_storage/test.py::test_create test_disk_access_storage/test.py::test_drop [gw9] [ 22%] PASSED test_attach_table_normalizer/test.py::test_attach_substr test_attach_table_normalizer/test.py::test_attach_substr_restart [gw5] [ 23%] PASSED test_dictionaries_update_and_reload/test.py::test_reload_after_fail_in_cache_dictionary test_dictionaries_update_and_reload/test.py::test_reload_after_loading [gw9] [ 24%] PASSED test_attach_table_normalizer/test.py::test_attach_substr_restart test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[0] [gw3] [ 25%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active [gw1] [ 26%] PASSED test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node1] test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node0] [gw7] [ 27%] PASSED test_disk_access_storage/test.py::test_drop test_create_query_constraints/test.py::test_create_query_const_constraints [gw5] [ 28%] PASSED test_dictionaries_update_and_reload/test.py::test_reload_after_loading test_dictionaries_update_and_reload/test.py::test_reload_while_loading [gw9] [ 29%] SKIPPED test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async [gw0] [ 30%] PASSED test_backup_restore_s3/test.py::test_backup_restore_system_tables_with_plain_rewritable_disk test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_local] [gw1] [ 31%] PASSED test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node0] [gw5] [ 32%] PASSED test_dictionaries_update_and_reload/test.py::test_reload_while_loading [gw0] [ 33%] PASSED test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_local] test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3] test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node1] [gw0] [ 34%] PASSED test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3] test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3_plain] [gw7] [ 35%] PASSED test_create_query_constraints/test.py::test_create_query_const_constraints test_create_query_constraints/test.py::test_create_query_minmax_constraints [gw0] [ 36%] PASSED test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3_plain] test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_local] [gw8] [ 37%] PASSED test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[0] test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[1048576] [gw7] [ 38%] PASSED test_create_query_constraints/test.py::test_create_query_minmax_constraints [gw0] [ 39%] PASSED test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_local] test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_s3_plain] [gw0] [ 40%] PASSED test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_s3_plain] test_backup_restore_s3/test.py::test_backup_to_s3 test_disks_app_other_disk_types/test.py::test_disks_app_plain_rewritable [gw9] [ 41%] PASSED test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async [gw3] [ 42%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] [gw1] [ 43%] PASSED test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node1] test_ddl_on_cluster_stop_waiting_for_offline_hosts/test.py::test_stop_waiting_for_offline_hosts [gw0] [ 44%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3 test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[auto_single] [gw8] [ 45%] PASSED test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[1048576] [gw0] [ 46%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[auto_single] test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_multipart] test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node0] test_dictionary_ddl_on_cluster/test.py::test_dictionary_ddl_on_cluster [gw2] [ 47%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random [gw1] [ 48%] PASSED test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node0] [gw7] [ 49%] PASSED test_disks_app_other_disk_types/test.py::test_disks_app_plain_rewritable test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node1] [gw9] [ 50%] PASSED test_dictionary_ddl_on_cluster/test.py::test_dictionary_ddl_on_cluster [gw1] [ 51%] PASSED test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node1] test_deduplicated_attached_part_rename/test.py::test_deduplicated_attached_part_renamed_after_attach test_enable_user_name_access_type/test.py::test_startup_scripts [gw3] [ 52%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node0] [gw0] [ 53%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_multipart] test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_single] [gw0] [ 54%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_single] test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_multipart] [gw5] [ 55%] PASSED test_ddl_on_cluster_stop_waiting_for_offline_hosts/test.py::test_stop_waiting_for_offline_hosts [gw3] [ 56%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[cache] test_distributed_config/test.py::test_distibuted_settings [gw8] [ 57%] PASSED test_deduplicated_attached_part_rename/test.py::test_deduplicated_attached_part_renamed_after_attach [gw1] [ 58%] PASSED test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node0] [gw9] [ 59%] PASSED test_enable_user_name_access_type/test.py::test_startup_scripts test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node1] [gw5] [ 60%] PASSED test_distributed_config/test.py::test_distibuted_settings [gw1] [ 61%] PASSED test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node1] [gw0] [ 62%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_multipart] test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_single] [gw0] [ 63%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_single] test_backup_restore_s3/test.py::test_backup_to_s3_multipart test_dictionaries_dependency/test.py::test_no_lazy_load [gw4] [ 64%] PASSED test_backup_restore_new/test_cancel_backup.py::test_cancel_backup [gw3] [ 65%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[cache] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[direct] test_backup_restore_new/test_cancel_backup.py::test_shutdown_cancel_backup [gw0] [ 66%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_multipart test_backup_restore_s3/test.py::test_backup_to_s3_named_collection [gw0] [ 67%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_named_collection test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3] [gw0] [ 68%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3] test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_other_bucket] [gw0] [ 69%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_other_bucket] test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_plain_rewritable] [gw2] [ 70%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance [gw4] [ 71%] PASSED test_backup_restore_new/test_cancel_backup.py::test_shutdown_cancel_backup test_backup_restore_on_cluster/test_different_versions.py::test_different_versions [gw0] [ 72%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_plain_rewritable] test_backup_restore_s3/test.py::test_backup_to_s3_native_copy_multipart [gw0] [ 73%] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_native_copy_multipart test_backup_restore_s3/test.py::test_backup_to_tar [gw0] [ 74%] PASSED test_backup_restore_s3/test.py::test_backup_to_tar test_backup_restore_s3/test.py::test_backup_to_tar_bz2 [gw0] [ 75%] PASSED test_backup_restore_s3/test.py::test_backup_to_tar_bz2 test_backup_restore_s3/test.py::test_backup_to_tar_gz [gw0] [ 76%] PASSED test_backup_restore_s3/test.py::test_backup_to_tar_gz test_backup_restore_s3/test.py::test_backup_to_tar_lzma [gw0] [ 77%] PASSED test_backup_restore_s3/test.py::test_backup_to_tar_lzma test_backup_restore_s3/test.py::test_backup_to_tar_xz [gw0] [ 78%] PASSED test_backup_restore_s3/test.py::test_backup_to_tar_xz test_backup_restore_s3/test.py::test_backup_to_tar_zst [gw3] [ 79%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[direct] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[flat] [gw0] [ 80%] PASSED test_backup_restore_s3/test.py::test_backup_to_tar_zst test_backup_restore_s3/test.py::test_backup_to_zip [gw1] [ 81%] PASSED test_dictionaries_dependency/test.py::test_no_lazy_load [gw0] [ 82%] PASSED test_backup_restore_s3/test.py::test_backup_to_zip test_backup_restore_s3/test.py::test_backup_with_fs_cache[False-True-False] [gw0] [ 83%] PASSED test_backup_restore_s3/test.py::test_backup_with_fs_cache[False-True-False] test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-False-False] [gw0] [ 84%] PASSED test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-False-False] test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-False] [gw0] [ 85%] PASSED test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-False] test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-True] [gw0] [ 86%] PASSED test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-True] test_backup_restore_s3/test.py::test_incremental_backup_append_table_def [gw0] [ 87%] PASSED test_backup_restore_s3/test.py::test_incremental_backup_append_table_def test_backup_restore_s3/test.py::test_user_specific_auth [gw0] [ 88%] PASSED test_backup_restore_s3/test.py::test_user_specific_auth [gw3] [ 89%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[flat] test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[hashed] [gw2] [ 90%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance test_distributed_load_balancing/test.py::test_load_balancing_in_order [gw3] [ 91%] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[hashed] [gw2] [ 92%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname [gw4] [ 93%] FAILED test_backup_restore_on_cluster/test_different_versions.py::test_different_versions [gw2] [ 94%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] [gw2] [ 95%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] [gw2] [ 96%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] test_distributed_load_balancing/test.py::test_load_balancing_round_robin [gw2] [ 97%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_round_robin [gw6] [ 98%] ERROR test_database_delta/test.py::test_complex_table_schema test_database_delta/test.py::test_embedded_database_and_tables [gw6] [ 99%] ERROR test_database_delta/test.py::test_embedded_database_and_tables test_database_delta/test.py::test_multiple_schemes_tables [gw6] [100%] ERROR test_database_delta/test.py::test_multiple_schemes_tables ==================================== ERRORS ==================================== _________________ ERROR at setup of test_complex_table_schema __________________ [gw6] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(scope="module") def started_cluster(): try: cluster = ClickHouseCluster(__file__) cluster.add_instance( "node1", main_configs=[], user_configs=[], image="clickhouse/integration-test-with-unity-catalog", with_installed_binary=False, tag=os.environ.get("DOCKER_BASE_WITH_UNITY_CATALOG_TAG", "latest") ) logging.info("Starting cluster...") > cluster.start() test_database_delta/test.py:43: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:2779: in start retry(log_function=logging_pulling_images)(run_and_check, images_pull_cmd) helpers/retry_decorator.py:20: in inner func(*args, **kwargs) helpers/cluster.py:132: in run_and_check res = subprocess.run( /usr/lib/python3.10/subprocess.py:505: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib/python3.10/subprocess.py:1154: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib/python3.10/subprocess.py:2021: in _communicate ready = selector.select(timeout) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , timeout = 5680 def select(self, timeout=None): # This is shared between poll() and epoll(). # epoll() has a different signature and handling of timeout parameter. if timeout is None: timeout = None elif timeout <= 0: timeout = 0 else: # poll() has a resolution of 1 millisecond, round away from # zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) ready = [] try: > fd_event_list = self._selector.poll(timeout) E Failed: Timeout >900.0s /usr/lib/python3.10/selectors.py:416: Failed ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ~~~~~~~~~~~~~~~~~~~~~ Stack of (140039534716480) ~~~~~~~~~~~~~~~~~~~~~ File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 411, in _perform_spawn reply.run() File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 341, in run self._result = func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 1160, in _thread_receiver msg = Message.from_io(io) File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 567, in from_io header = io.read(9) # type 1, channel 4, payload 4 File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 534, in read data = self._read(numbytes - len(buf)) ------------------------------ Captured log setup ------------------------------ 2026-10-29 20:20:43.882000 [ 643 ] DEBUG : Command:[docker ps | wc -l] (cluster.py:121, run_and_check) 2026-10-29 20:20:43.910000 [ 643 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-10-29 20:20:43.911000 [ 643 ] DEBUG : No running containers (conftest.py:95, cleanup_environment) 2026-10-29 20:20:43.911000 [ 643 ] DEBUG : Pruning Docker networks (conftest.py:97, cleanup_environment) 2026-10-29 20:20:43.911000 [ 643 ] DEBUG : Command:[docker network prune --force] (cluster.py:121, run_and_check) 2026-10-29 20:20:43.941000 [ 643 ] DEBUG : Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] (cluster.py:121, run_and_check) 2026-10-29 20:20:43.944000 [ 643 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:145, run_and_check) 2026-10-29 20:20:43.945000 [ 643 ] DEBUG : ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 (cluster.py:424, __init__) 2026-10-29 20:20:43.945000 [ 643 ] DEBUG : ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:424, __init__) 2026-10-29 20:20:43.945000 [ 643 ] DEBUG : ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:424, __init__) 2026-10-29 20:20:43.945000 [ 643 ] DEBUG : ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:424, __init__) 2026-10-29 20:20:43.945000 [ 643 ] DEBUG : ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:424, __init__) 2026-10-29 20:20:43.945000 [ 643 ] DEBUG : ENV HOSTNAME df81cdb70397 (cluster.py:424, __init__) 2026-10-29 20:20:43.945000 [ 643 ] DEBUG : ENV SHLVL 0 (cluster.py:424, __init__) 2026-10-29 20:20:43.945000 [ 643 ] DEBUG : ENV HOME /root (cluster.py:424, __init__) 2026-10-29 20:20:43.945000 [ 643 ] DEBUG : ENV OLDPWD / (cluster.py:424, __init__) 2026-10-29 20:20:43.945000 [ 643 ] DEBUG : ENV DOCKER_HELPER_TAG 5dc43a6382f0 (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV PYTHONUNBUFFERED 1 (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV DOCKER_PYTHON_BOTTLE_TAG d862517635bf (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV UBSAN_OPTIONS print_stacktrace=1 (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel0_0.jsonl --report-log-exclude-logs-on-passed-tests test_allow_feature_tier/test.py::test_allow_feature_tier_in_general_settings test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings_with_old_compatibility test_allow_feature_tier/test.py::test_allow_feature_tier_in_user test_allow_feature_tier/test.py::test_it_is_possible_to_enable_experimental_settings_in_default_profile test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active test_attach_table_normalizer/test.py::test_attach_substr test_attach_table_normalizer/test.py::test_attach_substr_restart 'test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[0]' 'test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[1048576]' test_backup_restore_new/test_cancel_backup.py::test_cancel_backup test_backup_restore_new/test_cancel_backup.py::test_shutdown_cancel_backup test_backup_restore_on_cluster/test_different_versions.py::test_different_versions test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async 'test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3]' 'test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3_other_bucket]' test_backup_restore_s3/test.py::test_backup_restore_system_tables_with_plain_rewritable_disk 'test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_local]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3_plain]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_local]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_s3_plain]' test_backup_restore_s3/test.py::test_backup_to_s3 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[auto_single]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_multipart]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_single]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_multipart]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_single]' test_backup_restore_s3/test.py::test_backup_to_s3_multipart test_backup_restore_s3/test.py::test_backup_to_s3_named_collection 'test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3]' 'test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_other_bucket]' 'test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_plain_rewritable]' test_backup_restore_s3/test.py::test_backup_to_s3_native_copy_multipart test_backup_restore_s3/test.py::test_backup_to_tar test_backup_restore_s3/test.py::test_backup_to_tar_bz2 test_backup_restore_s3/test.py::test_backup_to_tar_gz test_backup_restore_s3/test.py::test_backup_to_tar_lzma test_backup_restore_s3/test.py::test_backup_to_tar_xz test_backup_restore_s3/test.py::test_backup_to_tar_zst test_backup_restore_s3/test.py::test_backup_to_zip 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[False-True-False]' 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-False-False]' 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-False]' 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-True]' test_backup_restore_s3/test.py::test_incremental_backup_append_table_def test_backup_restore_s3/test.py::test_user_specific_auth test_create_query_constraints/test.py::test_create_query_const_constraints test_create_query_constraints/test.py::test_create_query_minmax_constraints test_database_delta/test.py::test_complex_table_schema test_database_delta/test.py::test_embedded_database_and_tables test_database_delta/test.py::test_multiple_schemes_tables test_ddl_on_cluster_stop_waiting_for_offline_hosts/test.py::test_stop_waiting_for_offline_hosts test_deduplicated_attached_part_rename/test.py::test_deduplicated_attached_part_renamed_after_attach 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[cache]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[direct]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[flat]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[hashed]' 'test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node0]' 'test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node1]' 'test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node0]' 'test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node1]' 'test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node0]' 'test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node1]' 'test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node0]' 'test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node1]' test_dictionaries_dependency/test.py::test_no_lazy_load test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_system_reload test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_timer test_dictionaries_update_and_reload/test.py::test_reload_after_fail_in_cache_dictionary test_dictionaries_update_and_reload/test.py::test_reload_after_loading test_dictionaries_update_and_reload/test.py::test_reload_while_loading test_dictionary_ddl_on_cluster/test.py::test_dictionary_ddl_on_cluster test_disk_access_storage/test.py::test_alter test_disk_access_storage/test.py::test_create test_disk_access_storage/test.py::test_drop test_disks_app_other_disk_types/test.py::test_disks_app_plain_rewritable test_distributed_config/test.py::test_distibuted_settings test_distributed_ddl_on_cross_replication/test.py::test_alter_ddl test_distributed_ddl_on_cross_replication/test.py::test_atomic_database test_distributed_ddl_on_cross_replication/test.py::test_non_query_with_table_ddl test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority]' 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative]' test_distributed_load_balancing/test.py::test_load_balancing_round_robin test_enable_user_name_access_type/test.py::test_startup_scripts test_encrypted_disk/test.py::test_add_keys test_encrypted_disk/test.py::test_add_keys_with_id 'test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False]' 'test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True]' 'test_encrypted_disk/test.py::test_backup_restore[File-local_policy-encrypted_policy-False]' -vvv (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV COMPOSE_HTTP_TIMEOUT 600 (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV DOCKER_KERBERIZED_HADOOP_TAG latest (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV DOCKER_CHANNEL stable (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV DOCKER_CLIENT_TIMEOUT 300 (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 (cluster.py:424, __init__) 2026-10-29 20:20:43.946000 [ 643 ] DEBUG : ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV PWD /ClickHouse/tests/integration (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV TZ Etc/UTC (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV DOCKER_BASE_TAG 5ccda723c1fc (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV LC_CTYPE C.UTF-8 (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV INTEGRATION_TESTS_RUN_ID 0 (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV WORKER_FREE_PORTS 30300 30301 30302 30303 30304 30305 30306 30307 30308 30309 30310 30311 30312 30313 30314 30315 30316 30317 30318 30319 30320 30321 30322 30323 30324 30325 30326 30327 30328 30329 30330 30331 30332 30333 30334 30335 30336 30337 30338 30339 30340 30341 30342 30343 30344 30345 30346 30347 30348 30349 (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV PYTEST_XDIST_TESTRUNUID 6a66ebd97c7b4ab49123ca04dba6586f (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV PYTEST_XDIST_WORKER gw6 (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV PYTEST_XDIST_WORKER_COUNT 10 (cluster.py:424, __init__) 2026-10-29 20:20:43.947000 [ 643 ] DEBUG : ENV PYTEST_CURRENT_TEST test_database_delta/test.py::test_complex_table_schema (setup) (cluster.py:424, __init__) 2026-10-29 20:20:43.948000 [ 643 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:724, __init__) 2026-10-29 20:20:43.948000 [ 643 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1662, add_instance) 2026-10-29 20:20:43.948000 [ 643 ] DEBUG : Cluster name: project_name:roottestdatabasedelta-gw6. Added instance name:node1 tag:latest base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/.env', '--project-name', 'roottestdatabasedelta-gw6', '--file', '/ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/node1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:1948, add_instance) 2026-10-29 20:20:43.948000 [ 643 ] INFO : Starting cluster... (test.py:42, started_cluster) 2026-10-29 20:20:43.948000 [ 643 ] INFO : Running tests in /ClickHouse/tests/integration/test_database_delta/test.py (cluster.py:2738, start) 2026-10-29 20:20:43.949000 [ 643 ] DEBUG : Cluster start called. is_up=False (cluster.py:2745, start) 2026-10-29 20:20:43.980000 [ 643 ] DEBUG : Docker networks for project roottestdatabasedelta-gw6 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 20:20:44.000000 [ 643 ] DEBUG : Docker containers for project roottestdatabasedelta-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 20:20:44.019000 [ 643 ] DEBUG : Docker volumes for project roottestdatabasedelta-gw6 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 20:20:44.019000 [ 643 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 20:20:44.040000 [ 643 ] DEBUG : Docker networks for project roottestdatabasedelta-gw6 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 20:20:44.063000 [ 643 ] DEBUG : Docker containers for project roottestdatabasedelta-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 20:20:44.083000 [ 643 ] DEBUG : Docker volumes for project roottestdatabasedelta-gw6 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 20:20:44.083000 [ 643 ] DEBUG : Command:[docker container list --all --filter name='^/roottestdatabasedelta-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 20:20:44.101000 [ 643 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 20:20:44.101000 [ 643 ] DEBUG : No running containers for project: roottestdatabasedelta-gw6 (cluster.py:879, cleanup) 2026-10-29 20:20:44.101000 [ 643 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 20:20:44.118000 [ 643 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 20:20:44.119000 [ 643 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 20:20:44.138000 [ 643 ] DEBUG : Stderr:Error response from daemon: a prune operation is already running (cluster.py:147, run_and_check) 2026-10-29 20:20:44.138000 [ 643 ] DEBUG : Exitcode:1 (cluster.py:149, run_and_check) 2026-10-29 20:20:44.138000 [ 643 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 20:20:44.139000 [ 643 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 20:20:44.154000 [ 643 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-10-29 20:20:44.155000 [ 643 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) 2026-10-29 20:20:44.155000 [ 643 ] DEBUG : Setup directory for instance: node1 (cluster.py:2758, start) 2026-10-29 20:20:44.155000 [ 643 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 20:20:44.156000 [ 643 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 20:20:44.156000 [ 643 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 20:20:44.156000 [ 643 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 20:20:44.156000 [ 643 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/node1/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 20:20:44.157000 [ 643 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/node1/database (cluster.py:4758, create_dir) 2026-10-29 20:20:44.157000 [ 643 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/node1/logs (cluster.py:4769, create_dir) 2026-10-29 20:20:44.157000 [ 643 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 20:20:44.157000 [ 643 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/.env (cluster.py:96, _create_env_file) 2026-10-29 20:20:44.157000 [ 643 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 20:20:44.157000 [ 643 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 20:20:44.157000 [ 643 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 20:20:44.157000 [ 643 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 20:20:44.171000 [ 643 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2026-10-29 20:20:44.171000 [ 643 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/.env --project-name roottestdatabasedelta-gw6 --file /ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/node1/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-10-29 20:25:44.182000 [ 643 ] INFO : Got exception pulling images: Command '['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/.env', '--project-name', 'roottestdatabasedelta-gw6', '--file', '/ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/node1/docker-compose.yml', 'pull']' timed out after 300 seconds (cluster.py:2775, logging_pulling_images) 2026-10-29 20:25:46.612000 [ 643 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/.env --project-name roottestdatabasedelta-gw6 --file /ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/node1/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-10-29 20:30:46.629000 [ 643 ] INFO : Got exception pulling images: Command '['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/.env', '--project-name', 'roottestdatabasedelta-gw6', '--file', '/ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/node1/docker-compose.yml', 'pull']' timed out after 300 seconds (cluster.py:2775, logging_pulling_images) 2026-10-29 20:30:49.519000 [ 643 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/.env --project-name roottestdatabasedelta-gw6 --file /ClickHouse/tests/integration/test_database_delta/_instances-0-gw6/node1/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-10-29 20:35:43.897000 [ 643 ] DEBUG : Failed to start cluster: (cluster.py:3169, start) 2026-10-29 20:35:43.897000 [ 643 ] DEBUG : Timeout >900.0s (cluster.py:3170, start) 2026-10-29 20:35:43.907000 [ 643 ] DEBUG : Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2779, in start retry(log_function=logging_pulling_images)(run_and_check, images_pull_cmd) File "/ClickHouse/tests/integration/helpers/retry_decorator.py", line 20, in inner func(*args, **kwargs) File "/ClickHouse/tests/integration/helpers/cluster.py", line 132, in run_and_check res = subprocess.run( File "/usr/lib/python3.10/subprocess.py", line 505, in run stdout, stderr = process.communicate(input, timeout=timeout) File "/usr/lib/python3.10/subprocess.py", line 1154, in communicate stdout, stderr = self._communicate(input, endtime, timeout) File "/usr/lib/python3.10/subprocess.py", line 2021, in _communicate ready = selector.select(timeout) File "/usr/lib/python3.10/selectors.py", line 416, in select fd_event_list = self._selector.poll(timeout) File "/usr/local/lib/python3.10/dist-packages/pytest_timeout.py", line 313, in handler timeout_sigalrm(item, settings) File "/usr/local/lib/python3.10/dist-packages/pytest_timeout.py", line 498, in timeout_sigalrm pytest.fail("Timeout >%ss" % settings.timeout) File "/usr/local/lib/python3.10/dist-packages/_pytest/outcomes.py", line 198, in fail raise Failed(msg=reason, pytrace=pytrace) Failed: Timeout >900.0s (cluster.py:3171, start) 2026-10-29 20:35:43.910000 [ 643 ] WARNING : docker compose up was not called. Trying to export docker.log for running containers (cluster.py:3252, shutdown) 2026-10-29 20:35:43.911000 [ 643 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 20:35:43.941000 [ 643 ] DEBUG : Docker networks for project roottestdatabasedelta-gw6 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 20:35:43.977000 [ 643 ] DEBUG : Docker containers for project roottestdatabasedelta-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 20:35:44.011000 [ 643 ] DEBUG : Docker volumes for project roottestdatabasedelta-gw6 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 20:35:44.011000 [ 643 ] DEBUG : Command:[docker container list --all --filter name='^/roottestdatabasedelta-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 20:35:44.045000 [ 643 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 20:35:44.045000 [ 643 ] DEBUG : No running containers for project: roottestdatabasedelta-gw6 (cluster.py:879, cleanup) 2026-10-29 20:35:44.045000 [ 643 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 20:35:44.078000 [ 643 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 20:35:44.078000 [ 643 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 20:35:45.280000 [ 643 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 20:35:45.281000 [ 643 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 20:35:45.281000 [ 643 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 20:35:45.281000 [ 643 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 20:35:45.313000 [ 643 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-10-29 20:35:45.314000 [ 643 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) 2026-10-29 20:35:45.316000 [ 643 ] WARNING : docker compose up was not called. Trying to export docker.log for running containers (cluster.py:3252, shutdown) 2026-10-29 20:35:45.317000 [ 643 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 20:35:45.340000 [ 643 ] DEBUG : Docker networks for project roottestdatabasedelta-gw6 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 20:35:45.369000 [ 643 ] DEBUG : Docker containers for project roottestdatabasedelta-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 20:35:45.399000 [ 643 ] DEBUG : Docker volumes for project roottestdatabasedelta-gw6 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 20:35:45.399000 [ 643 ] DEBUG : Command:[docker container list --all --filter name='^/roottestdatabasedelta-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 20:35:45.426000 [ 643 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 20:35:45.426000 [ 643 ] DEBUG : No running containers for project: roottestdatabasedelta-gw6 (cluster.py:879, cleanup) 2026-10-29 20:35:45.426000 [ 643 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 20:35:45.461000 [ 643 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 20:35:45.461000 [ 643 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 20:35:45.507000 [ 643 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 20:35:45.507000 [ 643 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 20:35:45.508000 [ 643 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 20:35:45.508000 [ 643 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 20:35:45.541000 [ 643 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-10-29 20:35:45.541000 [ 643 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) _____________ ERROR at setup of test_embedded_database_and_tables ______________ [gw6] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(scope="module") def started_cluster(): try: cluster = ClickHouseCluster(__file__) cluster.add_instance( "node1", main_configs=[], user_configs=[], image="clickhouse/integration-test-with-unity-catalog", with_installed_binary=False, tag=os.environ.get("DOCKER_BASE_WITH_UNITY_CATALOG_TAG", "latest") ) logging.info("Starting cluster...") > cluster.start() test_database_delta/test.py:43: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:2779: in start retry(log_function=logging_pulling_images)(run_and_check, images_pull_cmd) helpers/retry_decorator.py:20: in inner func(*args, **kwargs) helpers/cluster.py:132: in run_and_check res = subprocess.run( /usr/lib/python3.10/subprocess.py:505: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib/python3.10/subprocess.py:1154: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib/python3.10/subprocess.py:2021: in _communicate ready = selector.select(timeout) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , timeout = 5680 def select(self, timeout=None): # This is shared between poll() and epoll(). # epoll() has a different signature and handling of timeout parameter. if timeout is None: timeout = None elif timeout <= 0: timeout = 0 else: # poll() has a resolution of 1 millisecond, round away from # zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) ready = [] try: > fd_event_list = self._selector.poll(timeout) E Failed: Timeout >900.0s /usr/lib/python3.10/selectors.py:416: Failed ________________ ERROR at setup of test_multiple_schemes_tables ________________ [gw6] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(scope="module") def started_cluster(): try: cluster = ClickHouseCluster(__file__) cluster.add_instance( "node1", main_configs=[], user_configs=[], image="clickhouse/integration-test-with-unity-catalog", with_installed_binary=False, tag=os.environ.get("DOCKER_BASE_WITH_UNITY_CATALOG_TAG", "latest") ) logging.info("Starting cluster...") > cluster.start() test_database_delta/test.py:43: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:2779: in start retry(log_function=logging_pulling_images)(run_and_check, images_pull_cmd) helpers/retry_decorator.py:20: in inner func(*args, **kwargs) helpers/cluster.py:132: in run_and_check res = subprocess.run( /usr/lib/python3.10/subprocess.py:505: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib/python3.10/subprocess.py:1154: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib/python3.10/subprocess.py:2021: in _communicate ready = selector.select(timeout) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , timeout = 5680 def select(self, timeout=None): # This is shared between poll() and epoll(). # epoll() has a different signature and handling of timeout parameter. if timeout is None: timeout = None elif timeout <= 0: timeout = 0 else: # poll() has a resolution of 1 millisecond, round away from # zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) ready = [] try: > fd_event_list = self._selector.poll(timeout) E Failed: Timeout >900.0s /usr/lib/python3.10/selectors.py:416: Failed =================================== FAILURES =================================== ___________________________ test_different_versions ____________________________ [gw4] linux -- Python 3.10.12 /usr/bin/python3 def test_different_versions(): new_node.query( "CREATE TABLE tbl" " ON CLUSTER 'cluster_ver'" " (x UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')" " ORDER BY tuple()" ) new_node.query(f"INSERT INTO tbl VALUES (1)") old_node.query(f"INSERT INTO tbl VALUES (2)") backup_name = new_backup_name() initiator = random_node() print(f"Using {get_node_name(initiator)} as initiator for BACKUP") initiator.query(f"BACKUP TABLE tbl ON CLUSTER 'cluster_ver' TO {backup_name}") new_node.query("DROP TABLE tbl ON CLUSTER 'cluster_ver' SYNC") initiator = random_node() print(f"Using {get_node_name(initiator)} as initiator for RESTORE") initiator.query(f"RESTORE TABLE tbl ON CLUSTER 'cluster_ver' FROM {backup_name}") new_node.query("SYSTEM SYNC REPLICA ON CLUSTER 'cluster_ver' tbl") assert new_node.query("SELECT * FROM tbl ORDER BY x") == TSV([1, 2]) assert old_node.query("SELECT * FROM tbl ORDER BY x") == TSV([1, 2]) # Error NO_ELEMENTS_IN_CONFIG is unrelated. > assert ( new_node.query( "SELECT name, last_error_message FROM system.errors WHERE NOT (" "(name == 'NO_ELEMENTS_IN_CONFIG')" ")" ) == "" ) E assert "NETLINK_ERROR\tCan\\'t receive Netlink response: error -2\n" == '' E + NETLINK_ERROR Can\'t receive Netlink response: error -2 test_backup_restore_on_cluster/test_different_versions.py:105: AssertionError ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2026-10-29 20:23:57.042000 [ 607 ] INFO : Running tests in /ClickHouse/tests/integration/test_backup_restore_on_cluster/test_different_versions.py (cluster.py:2738, start) 2026-10-29 20:23:57.042000 [ 607 ] DEBUG : Cluster start called. is_up=False (cluster.py:2745, start) 2026-10-29 20:23:57.059000 [ 607 ] DEBUG : Docker networks for project roottestbackuprestoreonclusterdifferentversions-gw4 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 20:23:57.077000 [ 607 ] DEBUG : Docker containers for project roottestbackuprestoreonclusterdifferentversions-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 20:23:57.094000 [ 607 ] DEBUG : Docker volumes for project roottestbackuprestoreonclusterdifferentversions-gw4 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 20:23:57.094000 [ 607 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 20:23:57.113000 [ 607 ] DEBUG : Docker networks for project roottestbackuprestoreonclusterdifferentversions-gw4 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 20:23:57.136000 [ 607 ] DEBUG : Docker containers for project roottestbackuprestoreonclusterdifferentversions-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 20:23:57.162000 [ 607 ] DEBUG : Docker volumes for project roottestbackuprestoreonclusterdifferentversions-gw4 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 20:23:57.162000 [ 607 ] DEBUG : Command:[docker container list --all --filter name='^/roottestbackuprestoreonclusterdifferentversions-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 20:23:57.187000 [ 607 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 20:23:57.188000 [ 607 ] DEBUG : No running containers for project: roottestbackuprestoreonclusterdifferentversions-gw4 (cluster.py:879, cleanup) 2026-10-29 20:23:57.188000 [ 607 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 20:23:57.210000 [ 607 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 20:23:57.210000 [ 607 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 20:23:57.238000 [ 607 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 20:23:57.238000 [ 607 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 20:23:57.238000 [ 607 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 20:23:57.238000 [ 607 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 20:23:57.255000 [ 607 ] DEBUG : Stdout:3 (cluster.py:145, run_and_check) 2026-10-29 20:23:57.255000 [ 607 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-10-29 20:23:57.272000 [ 607 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 20:23:57.272000 [ 607 ] DEBUG : Volumes pruned: 3 (cluster.py:915, cleanup) 2026-10-29 20:23:57.272000 [ 607 ] DEBUG : Setup directory for instance: new_node (cluster.py:2758, start) 2026-10-29 20:23:57.272000 [ 607 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 20:23:57.272000 [ 607 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 20:23:57.273000 [ 607 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 20:23:57.273000 [ 607 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 20:23:57.273000 [ 607 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/backups_disk.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/cluster_different_versions.xml'] to /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/new_node/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 20:23:57.274000 [ 607 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/new_node/database (cluster.py:4758, create_dir) 2026-10-29 20:23:57.274000 [ 607 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/new_node/logs (cluster.py:4769, create_dir) 2026-10-29 20:23:57.274000 [ 607 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 20:23:57.274000 [ 607 ] INFO : external_dir_abs_path=/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/backups (cluster.py:4879, create_dir) 2026-10-29 20:23:57.274000 [ 607 ] DEBUG : Setup directory for instance: old_node (cluster.py:2758, start) 2026-10-29 20:23:57.275000 [ 607 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 20:23:57.275000 [ 607 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 20:23:57.275000 [ 607 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 20:23:57.275000 [ 607 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 20:23:57.275000 [ 607 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/backups_disk.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/cluster_different_versions.xml'] to /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/old_node/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 20:23:57.276000 [ 607 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/old_node/database (cluster.py:4758, create_dir) 2026-10-29 20:23:57.276000 [ 607 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/old_node/logs (cluster.py:4769, create_dir) 2026-10-29 20:23:57.276000 [ 607 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 20:23:57.276000 [ 607 ] INFO : external_dir_abs_path=/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/backups (cluster.py:4879, create_dir) 2026-10-29 20:23:57.276000 [ 607 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/.env (cluster.py:96, _create_env_file) 2026-10-29 20:23:57.276000 [ 607 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 20:23:57.276000 [ 607 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 20:23:57.276000 [ 607 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 20:23:57.277000 [ 607 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 20:23:57.283000 [ 607 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2026-10-29 20:23:57.284000 [ 607 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/.env --project-name roottestbackuprestoreonclusterdifferentversions-gw4 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/new_node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/old_node/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-10-29 20:27:54.918000 [ 607 ] DEBUG : Stderr: zoo1 Skipped - Image is already being pulled by new_node (cluster.py:147, run_and_check) 2026-10-29 20:27:54.918000 [ 607 ] DEBUG : Stderr: zoo2 Skipped - Image is already being pulled by new_node (cluster.py:147, run_and_check) 2026-10-29 20:27:54.919000 [ 607 ] DEBUG : Stderr: zoo3 Skipped - Image is already being pulled by new_node (cluster.py:147, run_and_check) 2026-10-29 20:27:54.919000 [ 607 ] DEBUG : Stderr: new_node Pulling (cluster.py:147, run_and_check) 2026-10-29 20:27:54.919000 [ 607 ] DEBUG : Stderr: old_node Pulling (cluster.py:147, run_and_check) 2026-10-29 20:27:54.919000 [ 607 ] DEBUG : Stderr: new_node Pulled (cluster.py:147, run_and_check) 2026-10-29 20:27:54.919000 [ 607 ] DEBUG : Stderr: 3823320faa42 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.919000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.919000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.919000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.919000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.919000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: 3823320faa42 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.920000 [ 607 ] DEBUG : Stderr: 3823320faa42 Downloading [> ] 303.2kB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.921000 [ 607 ] DEBUG : Stderr: 3823320faa42 Downloading [===============================> ] 17.07MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.921000 [ 607 ] DEBUG : Stderr: 3823320faa42 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.921000 [ 607 ] DEBUG : Stderr: 3823320faa42 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.921000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Downloading [> ] 100.7kB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.922000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [> ] 294.9kB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.922000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.922000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.923000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Downloading [==================================================>] 32B/32B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.923000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 5 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.923000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [==========> ] 5.898MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.924000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [================> ] 9.142MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.924000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [=====================> ] 12.09MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.924000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [=================================> ] 18.58MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.924000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [=========================================> ] 22.71MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.925000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [==========================================> ] 23.3MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.925000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [===============================================> ] 26.25MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.925000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [=================================================> ] 27.13MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.925000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 4 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.926000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [==================================================>] 27.51MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.926000 [ 607 ] DEBUG : Stderr: 3823320faa42 Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.926000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [> ] 98.3kB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.927000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [===============> ] 2.654MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.927000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [==========================> ] 4.719MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.927000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [====================================> ] 6.488MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.928000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [=============================================> ] 8.061MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.928000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [==============================================> ] 8.258MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.928000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [===============================================> ] 8.356MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.928000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 3 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.929000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [================================================> ] 8.552MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.929000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [==================================================>] 8.803MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.929000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.929000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 2 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.930000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 1 second (cluster.py:147, run_and_check) 2026-10-29 20:27:54.930000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Downloading [==================================================>] 32B/32B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.930000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 10 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.931000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 9 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.931000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 8 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.931000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 7 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.931000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 6 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.931000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 5 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 4 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [> ] 532.6kB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [===========> ] 32.83MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [========================> ] 71.27MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=======================================> ] 115.3MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Downloading [=======> ] 3.646kB/25.48kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Downloading [==================================================>] 25.48kB/25.48kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.932000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Downloading [> ] 10.88kB/863.5kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Downloading [==================================================>] 116B/116B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Downloading [==================================================>] 368B/368B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Downloading [==================================================>] 3.112kB/3.112kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 3 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 2 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 1 second (cluster.py:147, run_and_check) 2026-10-29 20:27:54.933000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Downloading [==================================================>] 32B/32B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 15 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 14 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 13 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 12 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 11 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 10 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 9 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 8 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 7 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 6 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 5 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 4 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.934000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 3 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 2 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 1 second (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Downloading [==================================================>] 32B/32B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 20 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 19 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 18 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 17 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 16 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 15 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 14 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 13 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.935000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 12 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 11 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 10 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 9 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 8 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 7 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 6 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 5 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 4 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 3 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 2 seconds (cluster.py:147, run_and_check) 2026-10-29 20:27:54.936000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Retrying in 1 second (cluster.py:147, run_and_check) 2026-10-29 20:27:54.937000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Downloading [==================================================>] 32B/32B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.937000 [ 607 ] DEBUG : Stderr: 3823320faa42 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.937000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.937000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.938000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.938000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.938000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.938000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Pulling fs layer (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Waiting (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Downloading [==================================================>] 32B/32B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.939000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Downloading [> ] 96.97kB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 3823320faa42 Downloading [> ] 277.2kB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Downloading [================> ] 2.934MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 3823320faa42 Downloading [=======> ] 4.206MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 3823320faa42 Downloading [==================> ] 10.36MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 3823320faa42 Downloading [============================> ] 15.41MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 3823320faa42 Downloading [======================================> ] 21MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 3823320faa42 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 3823320faa42 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.940000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [> ] 294.9kB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [======> ] 3.539MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [===========> ] 6.488MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [================> ] 8.847MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [===================> ] 10.62MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [=========================> ] 14.16MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [==================================> ] 18.87MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [=========================================> ] 22.71MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [==========================================> ] 23.3MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [==============================================> ] 25.36MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [================================================> ] 26.54MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [=================================================> ] 27.13MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Extracting [==================================================>] 27.51MB/27.51MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 3823320faa42 Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.941000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [> ] 98.3kB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [======> ] 1.18MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [=====================> ] 3.834MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [===============================> ] 5.603MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [=========================================> ] 7.274MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [=============================================> ] 8.061MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [==============================================> ] 8.159MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [===============================================> ] 8.356MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [================================================> ] 8.454MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Extracting [==================================================>] 8.803MB/8.803MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 9992e46614b1 Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Extracting [==================================================>] 32B/32B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Extracting [==================================================>] 32B/32B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.942000 [ 607 ] DEBUG : Stderr: 4f4fb700ef54 Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.943000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Downloading [================================> ] 16.38kB/25.48kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.943000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Downloading [==================================================>] 25.48kB/25.48kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.943000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.943000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.943000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Downloading [> ] 15.74kB/863.5kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.943000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [> ] 539.3kB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.943000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Downloading [==================================================>] 863.5kB/863.5kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.943000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.943000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=> ] 3.225MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [===> ] 9.672MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=====> ] 16.08MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [========> ] 25.16MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [==========> ] 31.59MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=============> ] 38.01MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [===============> ] 43.91MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=================> ] 49.25MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [===================> ] 55.11MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=====================> ] 61MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=======================> ] 67.41MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=========================> ] 73.83MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.944000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [===========================> ] 79.72MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [==============================> ] 86.65MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [================================> ] 92.54MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=================================> ] 97.89MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [===================================> ] 103.3MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [======================================> ] 110.7MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [========================================> ] 117.7MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [===========================================> ] 124.6MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=============================================> ] 131MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [===============================================> ] 136.4MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Downloading [=================================================> ] 142.8MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [> ] 557.1kB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.945000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [==> ] 7.799MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [======> ] 17.83MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [=======> ] 22.84MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [==========> ] 28.97MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [===========> ] 33.42MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [============> ] 37.32MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [==============> ] 41.78MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [===============> ] 45.68MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [================> ] 48.46MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [=================> ] 51.81MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [===================> ] 55.15MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [====================> ] 58.49MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [=====================> ] 62.39MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.946000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [======================> ] 66.29MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [========================> ] 70.75MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [=========================> ] 74.09MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [===========================> ] 77.99MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [============================> ] 83MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [==============================> ] 88.01MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [================================> ] 93.59MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [==================================> ] 99.71MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [=====================================> ] 107.5MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [========================================> ] 116.4MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [===========================================> ] 124.2MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [=============================================> ] 132MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [===============================================> ] 137MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.947000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [================================================> ] 139.3MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [================================================> ] 140.4MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [=================================================> ] 141.5MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [=================================================> ] 142.6MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [=================================================> ] 143.7MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Extracting [==================================================>] 144.2MB/144.2MB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: cfd7763775f9 Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Extracting [==================================================>] 25.48kB/25.48kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Extracting [==================================================>] 25.48kB/25.48kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: a517e5e9f3cd Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Extracting [=> ] 32.77kB/863.5kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Extracting [==================================================>] 863.5kB/863.5kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Extracting [==================================================>] 863.5kB/863.5kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: 3ce37c9627a5 Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.948000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Downloading [==================================================>] 368B/368B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Downloading [==================================================>] 3.112kB/3.112kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Verifying Checksum (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Downloading [==================================================>] 116B/116B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Download complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Extracting [==================================================>] 116B/116B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Extracting [==================================================>] 116B/116B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 3f8a7f45c802 Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Extracting [==================================================>] 368B/368B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Extracting [==================================================>] 368B/368B (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: fd4d333a83bf Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Extracting [==================================================>] 3.112kB/3.112kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Extracting [==================================================>] 3.112kB/3.112kB (cluster.py:147, run_and_check) 2026-10-29 20:27:54.949000 [ 607 ] DEBUG : Stderr: 12e84a19c313 Pull complete (cluster.py:147, run_and_check) 2026-10-29 20:27:54.950000 [ 607 ] DEBUG : Stderr: old_node Pulled (cluster.py:147, run_and_check) 2026-10-29 20:27:54.950000 [ 607 ] DEBUG : Setup ZooKeeper (cluster.py:2799, start) 2026-10-29 20:27:54.950000 [ 607 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper1/log', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper1/config', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper1/coordination', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper2/log', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper2/config', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper2/coordination', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper3/log', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper3/config', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/keeper3/coordination'] (cluster.py:2800, start) 2026-10-29 20:27:54.952000 [ 607 ] DEBUG : Command:[docker compose --project-name roottestbackuprestoreonclusterdifferentversions-gw4 --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] (cluster.py:121, run_and_check) 2026-10-29 20:27:55.744000 [ 607 ] DEBUG : Stderr:time="2026-10-29T20:27:55Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:147, run_and_check) 2026-10-29 20:27:55.744000 [ 607 ] DEBUG : Stderr: Network roottestbackuprestoreonclusterdifferentversions-gw4_default Creating (cluster.py:147, run_and_check) 2026-10-29 20:27:55.744000 [ 607 ] DEBUG : Stderr: Network roottestbackuprestoreonclusterdifferentversions-gw4_default Created (cluster.py:147, run_and_check) 2026-10-29 20:27:55.744000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:27:55.744000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr:time="2026-10-29T20:27:55Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-10-29 20:27:55.745000 [ 607 ] DEBUG : Stderr:time="2026-10-29T20:27:55Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-10-29 20:27:55.746000 [ 607 ] DEBUG : Wait ZooKeeper to start (cluster.py:2436, wait_zookeeper_to_start) 2026-10-29 20:27:55.746000 [ 607 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2005, get_instance_ip) 2026-10-29 20:27:55.748000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:27:55.749000 [ 607 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.1.4, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-10-29 20:27:55.752000 [ 607 ] INFO : Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:27:55.752000 [ 607 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 20:27:55.844000 [ 607 ] INFO : Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:27:55.844000 [ 607 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 20:27:56.069000 [ 607 ] INFO : Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:27:56.069000 [ 607 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 20:27:56.637000 [ 607 ] INFO : Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:27:56.637000 [ 607 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 20:27:57.728000 [ 607 ] INFO : Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:27:57.729000 [ 607 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 20:28:00.427000 [ 607 ] INFO : Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:28:00.428000 [ 607 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-10-29 20:28:00.432000 [ 607 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-10-29 20:28:00.433000 [ 607 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-10-29 20:28:00.434000 [ 607 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-10-29 20:28:00.435000 [ 607 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-10-29 20:28:00.438000 [ 607 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-10-29 20:28:00.438000 [ 607 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-10-29 20:28:00.438000 [ 607 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-10-29 20:28:00.521000 [ 607 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-10-29 20:28:00.521000 [ 607 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-10-29 20:28:00.522000 [ 607 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:2005, get_instance_ip) 2026-10-29 20:28:00.525000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:00.526000 [ 607 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.1.2, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-10-29 20:28:00.528000 [ 607 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:28:00.529000 [ 607 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-10-29 20:28:00.538000 [ 607 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-10-29 20:28:00.538000 [ 607 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-10-29 20:28:00.539000 [ 607 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-10-29 20:28:00.540000 [ 607 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-10-29 20:28:00.544000 [ 607 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-10-29 20:28:00.544000 [ 607 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-10-29 20:28:00.544000 [ 607 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-10-29 20:28:00.644000 [ 607 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-10-29 20:28:00.644000 [ 607 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-10-29 20:28:00.645000 [ 607 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:2005, get_instance_ip) 2026-10-29 20:28:00.647000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:00.647000 [ 607 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.1.3, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-10-29 20:28:00.648000 [ 607 ] INFO : Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:28:00.649000 [ 607 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-10-29 20:28:00.657000 [ 607 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-10-29 20:28:00.658000 [ 607 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-10-29 20:28:00.659000 [ 607 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-10-29 20:28:00.660000 [ 607 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-10-29 20:28:00.663000 [ 607 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-10-29 20:28:00.663000 [ 607 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-10-29 20:28:00.664000 [ 607 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-10-29 20:28:00.764000 [ 607 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-10-29 20:28:00.764000 [ 607 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-10-29 20:28:00.765000 [ 607 ] DEBUG : All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') (cluster.py:2452, wait_zookeeper_nodes_to_start) 2026-10-29 20:28:00.765000 [ 607 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/.env --project-name roottestbackuprestoreonclusterdifferentversions-gw4 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/new_node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/old_node/docker-compose.yml up -d --no-recreate') (cluster.py:3139, start) 2026-10-29 20:28:00.765000 [ 607 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/.env --project-name roottestbackuprestoreonclusterdifferentversions-gw4 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/new_node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/old_node/docker-compose.yml up -d --no-recreate] (cluster.py:121, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Running (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Running (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Running (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : ClickHouse instance created (cluster.py:3147, start) 2026-10-29 20:28:01.353000 [ 607 ] DEBUG : get_instance_ip instance_name=new_node (cluster.py:2005, get_instance_ip) 2026-10-29 20:28:01.355000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:01.356000 [ 607 ] DEBUG : get_instance_ip instance_name=new_node (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 20:28:01.357000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:01.357000 [ 607 ] DEBUG : Waiting for ClickHouse start in new_node, ip: 172.16.1.6... (cluster.py:3155, start) 2026-10-29 20:28:01.359000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:01.360000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:01.463000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:01.566000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:01.670000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:01.774000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:01.877000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:01.981000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.085000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.189000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.292000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.395000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.498000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.602000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.706000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.810000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/c90933a473f03504b8a37e2eda6445a6ba8016941e5efaae689410107626774e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.811000 [ 607 ] DEBUG : ClickHouse new_node started (cluster.py:3159, start) 2026-10-29 20:28:02.812000 [ 607 ] DEBUG : get_instance_ip instance_name=old_node (cluster.py:2005, get_instance_ip) 2026-10-29 20:28:02.814000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.815000 [ 607 ] DEBUG : get_instance_ip instance_name=old_node (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 20:28:02.817000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.817000 [ 607 ] DEBUG : Waiting for ClickHouse start in old_node, ip: 172.16.1.5... (cluster.py:3155, start) 2026-10-29 20:28:02.820000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.823000 [ 607 ] DEBUG : http://localhost:None "GET /v1.46/containers/3cec390e3aacf1b5abe6a4c6d36450d6a06de98aed05754b399974e823ddc050/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:28:02.824000 [ 607 ] DEBUG : ClickHouse old_node started (cluster.py:3159, start) ----------------------------- Captured stdout call ----------------------------- Using old_node as initiator for BACKUP Using new_node as initiator for RESTORE ------------------------------ Captured log call ------------------------------- 2026-10-29 20:28:02.828000 [ 607 ] DEBUG : Executing query CREATE TABLE tbl ON CLUSTER 'cluster_ver' (x UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}') ORDER BY tuple() on new_node (cluster.py:3648, query) 2026-10-29 20:28:03.245000 [ 607 ] DEBUG : Executing query INSERT INTO tbl VALUES (1) on new_node (cluster.py:3648, query) 2026-10-29 20:28:03.511000 [ 607 ] DEBUG : Executing query INSERT INTO tbl VALUES (2) on old_node (cluster.py:3648, query) 2026-10-29 20:28:03.778000 [ 607 ] DEBUG : Executing query BACKUP TABLE tbl ON CLUSTER 'cluster_ver' TO Disk('backups', '1') on old_node (cluster.py:3648, query) 2026-10-29 20:28:04.346000 [ 607 ] DEBUG : Executing query DROP TABLE tbl ON CLUSTER 'cluster_ver' SYNC on new_node (cluster.py:3648, query) 2026-10-29 20:28:04.814000 [ 607 ] DEBUG : Executing query RESTORE TABLE tbl ON CLUSTER 'cluster_ver' FROM Disk('backups', '1') on new_node (cluster.py:3648, query) 2026-10-29 20:28:05.481000 [ 607 ] DEBUG : Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster_ver' tbl on new_node (cluster.py:3648, query) 2026-10-29 20:28:05.848000 [ 607 ] DEBUG : Executing query SELECT * FROM tbl ORDER BY x on new_node (cluster.py:3648, query) 2026-10-29 20:28:06.165000 [ 607 ] DEBUG : Executing query SELECT * FROM tbl ORDER BY x on old_node (cluster.py:3648, query) 2026-10-29 20:28:06.481000 [ 607 ] DEBUG : Executing query SELECT name, last_error_message FROM system.errors WHERE NOT ((name == 'NO_ELEMENTS_IN_CONFIG')) on new_node (cluster.py:3648, query) ---------------------------- Captured log teardown ----------------------------- 2026-10-29 20:28:06.844000 [ 607 ] DEBUG : Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster_ver' SYNC on new_node (cluster.py:3648, query) 2026-10-29 20:28:07.410000 [ 607 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/.env --project-name roottestbackuprestoreonclusterdifferentversions-gw4 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/new_node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/old_node/docker-compose.yml stop --timeout 20] (cluster.py:121, run_and_check) 2026-10-29 20:28:15.388000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:28:15.388000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:28:15.388000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:28:15.388000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:28:15.388000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:28:15.388000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:28:15.388000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:28:15.388000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:28:15.388000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:28:15.388000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:28:15.389000 [ 607 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/new_node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/new_node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 20:28:15.407000 [ 607 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/old_node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/old_node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 20:28:15.427000 [ 607 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/.env --project-name roottestbackuprestoreonclusterdifferentversions-gw4 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/new_node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-0-gw4/old_node/docker-compose.yml down --volumes] (cluster.py:121, run_and_check) 2026-10-29 20:28:16.217000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:28:16.217000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:28:16.217000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:28:16.217000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:28:16.217000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-new_node-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-old_node-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:28:16.218000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:28:16.219000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo3-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:28:16.219000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo2-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:28:16.219000 [ 607 ] DEBUG : Stderr: Container roottestbackuprestoreonclusterdifferentversions-gw4-zoo1-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:28:16.219000 [ 607 ] DEBUG : Stderr: Network roottestbackuprestoreonclusterdifferentversions-gw4_default Removing (cluster.py:147, run_and_check) 2026-10-29 20:28:16.219000 [ 607 ] DEBUG : Stderr: Network roottestbackuprestoreonclusterdifferentversions-gw4_default Removed (cluster.py:147, run_and_check) 2026-10-29 20:28:16.219000 [ 607 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 20:28:16.250000 [ 607 ] DEBUG : Docker networks for project roottestbackuprestoreonclusterdifferentversions-gw4 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 20:28:16.282000 [ 607 ] DEBUG : Docker containers for project roottestbackuprestoreonclusterdifferentversions-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 20:28:16.311000 [ 607 ] DEBUG : Docker volumes for project roottestbackuprestoreonclusterdifferentversions-gw4 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 20:28:16.312000 [ 607 ] DEBUG : Command:[docker container list --all --filter name='^/roottestbackuprestoreonclusterdifferentversions-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 20:28:16.343000 [ 607 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 20:28:16.343000 [ 607 ] DEBUG : No running containers for project: roottestbackuprestoreonclusterdifferentversions-gw4 (cluster.py:879, cleanup) 2026-10-29 20:28:16.343000 [ 607 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 20:28:16.368000 [ 607 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 20:28:16.368000 [ 607 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 20:28:16.392000 [ 607 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 20:28:16.392000 [ 607 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 20:28:16.392000 [ 607 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 20:28:16.392000 [ 607 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 20:28:16.418000 [ 607 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-10-29 20:28:16.418000 [ 607 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) ----------------- generated report log file: parallel0_0.jsonl ----------------- ============================== slowest durations =============================== 901.67s setup test_database_delta/test.py::test_complex_table_schema 245.79s setup test_backup_restore_on_cluster/test_different_versions.py::test_different_versions 130.90s call test_backup_restore_new/test_cancel_backup.py::test_cancel_backup 86.84s call test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 86.33s call test_distributed_load_balancing/test.py::test_load_balancing_in_order 85.15s call test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 85.11s call test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance 76.66s call test_distributed_load_balancing/test.py::test_load_balancing_default 50.12s call test_dictionaries_dependency/test.py::test_no_lazy_load 39.92s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[hashed] 39.61s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[direct] 39.05s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[flat] 36.78s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[cache] 36.28s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] 36.27s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] 35.46s call test_backup_restore_s3/test.py::test_backup_restore_system_tables_with_plain_rewritable_disk 35.29s call test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_multipart] 34.97s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] 32.71s call test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_multipart] 31.53s call test_ddl_on_cluster_stop_waiting_for_offline_hosts/test.py::test_stop_waiting_for_offline_hosts 31.05s teardown test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[1048576] 28.49s setup test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3] 25.87s teardown test_backup_restore_s3/test.py::test_user_specific_auth 25.00s call test_enable_user_name_access_type/test.py::test_startup_scripts 22.19s teardown test_encrypted_disk/test.py::test_backup_restore[File-local_policy-encrypted_policy-False] 21.92s teardown test_disks_app_other_disk_types/test.py::test_disks_app_plain_rewritable 20.36s setup test_disks_app_other_disk_types/test.py::test_disks_app_plain_rewritable 20.12s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] 19.58s setup test_distributed_ddl_on_cross_replication/test.py::test_alter_ddl 18.80s setup test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 18.27s call test_disk_access_storage/test.py::test_drop 17.68s setup test_deduplicated_attached_part_rename/test.py::test_deduplicated_attached_part_renamed_after_attach 17.54s setup test_encrypted_disk/test.py::test_add_keys 17.52s teardown test_dictionaries_update_and_reload/test.py::test_reload_while_loading 17.51s call test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node1] 17.38s setup test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node0] 17.10s call test_dictionaries_update_and_reload/test.py::test_reload_after_fail_in_cache_dictionary 17.01s call test_disk_access_storage/test.py::test_alter 16.83s call test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node0] 16.65s call test_dictionaries_update_and_reload/test.py::test_reload_after_loading 15.61s call test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node1] 15.37s setup test_disk_access_storage/test.py::test_alter 15.06s call test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] 15.06s setup test_ddl_on_cluster_stop_waiting_for_offline_hosts/test.py::test_stop_waiting_for_offline_hosts 14.92s setup test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[0] 14.72s setup test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_system_reload 14.67s call test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node0] 14.57s call test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_timer 13.02s setup test_allow_feature_tier/test.py::test_allow_feature_tier_in_general_settings 12.92s call test_backup_restore_s3/test.py::test_user_specific_auth 12.80s setup test_create_query_constraints/test.py::test_create_query_const_constraints 12.79s setup test_distributed_config/test.py::test_distibuted_settings 12.78s setup test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active 12.77s call test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[0] 12.71s setup test_attach_table_normalizer/test.py::test_attach_substr 12.59s call test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node1] 12.17s call test_dictionary_ddl_on_cluster/test.py::test_dictionary_ddl_on_cluster 11.99s setup test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async 11.94s call test_backup_restore_new/test_cancel_backup.py::test_shutdown_cancel_backup 11.38s call test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node0] 11.03s call test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[1048576] 10.95s setup test_dictionary_ddl_on_cluster/test.py::test_dictionary_ddl_on_cluster 10.71s call test_disk_access_storage/test.py::test_create 10.45s call test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings 10.06s call test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node0] 9.93s call test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node1] 9.58s teardown test_backup_restore_on_cluster/test_different_versions.py::test_different_versions 9.43s call test_allow_feature_tier/test.py::test_allow_feature_tier_in_user 8.76s call test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 8.56s call test_distributed_load_balancing/test.py::test_load_balancing_round_robin 8.49s call test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 8.46s call test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 8.44s call test_dictionaries_update_and_reload/test.py::test_reload_while_loading 8.07s teardown test_dictionaries_dependency/test.py::test_no_lazy_load 8.00s teardown test_distributed_ddl_on_cross_replication/test.py::test_non_query_with_table_ddl 7.61s call test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async 7.13s call test_backup_restore_s3/test.py::test_backup_to_s3_native_copy_multipart 6.90s teardown test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async 6.89s setup test_backup_restore_new/test_cancel_backup.py::test_cancel_backup 6.75s teardown test_distributed_load_balancing/test.py::test_load_balancing_round_robin 6.67s teardown test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node0] 6.61s call test_backup_restore_s3/test.py::test_backup_to_s3_multipart 6.43s teardown test_distributed_config/test.py::test_distibuted_settings 6.27s teardown test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node1] 6.08s teardown test_dictionary_ddl_on_cluster/test.py::test_dictionary_ddl_on_cluster 6.08s teardown test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node1] 6.07s teardown test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node0] 5.78s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[hashed] 5.73s teardown test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node1] 5.72s teardown test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node0] 5.68s teardown test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node0] 5.62s teardown test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node1] 5.18s teardown test_ddl_on_cluster_stop_waiting_for_offline_hosts/test.py::test_stop_waiting_for_offline_hosts 5.11s teardown test_create_query_constraints/test.py::test_create_query_minmax_constraints 4.82s call test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3_other_bucket] 4.78s call test_backup_restore_s3/test.py::test_incremental_backup_append_table_def 4.30s call test_distributed_ddl_on_cross_replication/test.py::test_atomic_database 4.30s call test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[auto_single] 4.23s call test_encrypted_disk/test.py::test_add_keys 4.20s call test_encrypted_disk/test.py::test_add_keys_with_id 4.05s call test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_single] 3.95s call test_attach_table_normalizer/test.py::test_attach_substr_restart 3.95s call test_backup_restore_s3/test.py::test_backup_to_s3 3.92s call test_backup_restore_on_cluster/test_different_versions.py::test_different_versions 3.87s call test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-False-False] 3.86s call test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_single] 3.83s call test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3] 3.77s call test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-True] 3.62s call test_backup_restore_s3/test.py::test_backup_with_fs_cache[False-True-False] 3.62s call test_backup_restore_s3/test.py::test_backup_to_tar_lzma 3.62s call test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-False] 3.61s call test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_plain_rewritable] 3.58s call test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_s3_plain] 3.56s call test_allow_feature_tier/test.py::test_allow_feature_tier_in_general_settings 3.54s call test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3] 3.53s call test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_other_bucket] 3.52s call test_backup_restore_s3/test.py::test_backup_to_tar_xz 3.48s call test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3] 3.48s call test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_system_reload 3.47s call test_backup_restore_s3/test.py::test_backup_to_tar_bz2 3.43s call test_deduplicated_attached_part_rename/test.py::test_deduplicated_attached_part_renamed_after_attach 3.42s call test_backup_restore_s3/test.py::test_backup_to_tar_zst 3.40s call test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_local] 3.27s call test_backup_restore_s3/test.py::test_backup_to_tar 3.23s teardown test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active 3.23s call test_backup_restore_s3/test.py::test_backup_to_zip 3.22s call test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3_plain] 3.17s call test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_local] 3.17s call test_backup_restore_s3/test.py::test_backup_to_s3_named_collection 3.05s call test_create_query_constraints/test.py::test_create_query_minmax_constraints 3.02s call test_backup_restore_s3/test.py::test_backup_to_tar_gz 2.61s call test_distributed_ddl_on_cross_replication/test.py::test_alter_ddl 2.57s teardown test_deduplicated_attached_part_rename/test.py::test_deduplicated_attached_part_renamed_after_attach 2.31s call test_create_query_constraints/test.py::test_create_query_const_constraints 2.24s setup test_disk_access_storage/test.py::test_create 2.14s teardown test_attach_table_normalizer/test.py::test_attach_substr_restart 2.04s call test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings_with_old_compatibility 1.98s call test_encrypted_disk/test.py::test_backup_restore[File-local_policy-encrypted_policy-False] 1.94s call test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] 1.76s call test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] 1.65s call test_allow_feature_tier/test.py::test_it_is_possible_to_enable_experimental_settings_in_default_profile 1.64s teardown test_backup_restore_new/test_cancel_backup.py::test_shutdown_cancel_backup 1.47s teardown test_disk_access_storage/test.py::test_drop 1.45s call test_distributed_config/test.py::test_distibuted_settings 1.39s teardown test_allow_feature_tier/test.py::test_it_is_possible_to_enable_experimental_settings_in_default_profile 1.11s call test_distributed_ddl_on_cross_replication/test.py::test_non_query_with_table_ddl 1.09s setup test_disk_access_storage/test.py::test_drop 1.00s setup test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 1.00s setup test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 1.00s setup test_distributed_load_balancing/test.py::test_load_balancing_round_robin 1.00s setup test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 0.95s setup test_distributed_load_balancing/test.py::test_load_balancing_in_order 0.95s setup test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 0.91s call test_attach_table_normalizer/test.py::test_attach_substr 0.85s setup test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance 0.80s setup test_distributed_load_balancing/test.py::test_load_balancing_default 0.52s call test_disks_app_other_disk_types/test.py::test_disks_app_plain_rewritable 0.27s call test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active 0.27s teardown test_encrypted_disk/test.py::test_add_keys 0.22s teardown test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] 0.22s teardown test_backup_restore_new/test_cancel_backup.py::test_cancel_backup 0.22s teardown test_encrypted_disk/test.py::test_add_keys_with_id 0.22s teardown test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] 0.01s teardown test_backup_restore_s3/test.py::test_backup_to_tar_lzma 0.00s setup test_create_query_constraints/test.py::test_create_query_minmax_constraints 0.00s setup test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] 0.00s teardown test_create_query_constraints/test.py::test_create_query_const_constraints 0.00s setup test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3_plain] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3] 0.00s setup test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-False] 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_multipart] 0.00s setup test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] 0.00s setup test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node1] 0.00s teardown test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3] 0.00s teardown test_disk_access_storage/test.py::test_alter 0.00s setup test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[1048576] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_single] 0.00s setup test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node1] 0.00s setup test_backup_restore_new/test_cancel_backup.py::test_shutdown_cancel_backup 0.00s teardown test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[0] 0.00s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[flat] 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_local] 0.00s setup test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_timer 0.00s setup test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-True] 0.00s teardown test_enable_user_name_access_type/test.py::test_startup_scripts 0.00s setup test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3_other_bucket] 0.00s setup test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings_with_old_compatibility 0.00s setup test_backup_restore_s3/test.py::test_backup_to_tar_xz 0.00s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[hashed] 0.00s teardown test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[auto_single] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_single] 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 0.00s setup test_encrypted_disk/test.py::test_backup_restore[File-local_policy-encrypted_policy-False] 0.00s setup test_allow_feature_tier/test.py::test_it_is_possible_to_enable_experimental_settings_in_default_profile 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_multipart] 0.00s setup test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node0] 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[direct] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_tar_lzma 0.00s teardown test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_timer 0.00s teardown test_distributed_ddl_on_cross_replication/test.py::test_alter_ddl 0.00s teardown test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-False-False] 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] 0.00s setup test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node0] 0.00s setup test_enable_user_name_access_type/test.py::test_startup_scripts 0.00s teardown test_allow_feature_tier/test.py::test_allow_feature_tier_in_user 0.00s setup test_backup_restore_s3/test.py::test_incremental_backup_append_table_def 0.00s setup test_backup_restore_s3/test.py::test_user_specific_auth 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_multipart] 0.00s teardown test_allow_feature_tier/test.py::test_allow_feature_tier_in_general_settings 0.00s teardown test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-False] 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_tar_gz 0.00s setup test_encrypted_disk/test.py::test_add_keys_with_id 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[flat] 0.00s setup test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings 0.00s teardown test_disk_access_storage/test.py::test_create 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_multipart] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3 0.00s setup test_backup_restore_s3/test.py::test_backup_to_tar_gz 0.00s setup test_dictionaries_update_and_reload/test.py::test_reload_after_loading 0.00s teardown test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-True] 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_default 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_local] 0.00s setup test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node1] 0.00s setup test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node1] 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 0.00s setup test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node0] 0.00s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[direct] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_multipart 0.00s teardown test_database_delta/test.py::test_complex_table_schema 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_s3_plain] 0.00s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] 0.00s teardown test_distributed_ddl_on_cross_replication/test.py::test_atomic_database 0.00s setup test_dictionaries_update_and_reload/test.py::test_reload_after_fail_in_cache_dictionary 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3 0.00s setup test_backup_restore_s3/test.py::test_backup_to_tar_bz2 0.00s teardown test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_system_reload 0.00s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_tar_bz2 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_single] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_tar_zst 0.00s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[cache] 0.00s setup test_backup_restore_s3/test.py::test_backup_with_fs_cache[False-True-False] 0.00s setup test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-False-False] 0.00s teardown test_database_delta/test.py::test_multiple_schemes_tables 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_other_bucket] 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[auto_single] 0.00s setup test_distributed_ddl_on_cross_replication/test.py::test_non_query_with_table_ddl 0.00s setup test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_s3_plain] 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] 0.00s setup test_dictionaries_dependency/test.py::test_no_lazy_load 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_tar 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_plain_rewritable] 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_single] 0.00s teardown test_backup_restore_s3/test.py::test_incremental_backup_append_table_def 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[cache] 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_native_copy_multipart 0.00s setup test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_local] 0.00s teardown test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3] 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3_plain] 0.00s setup test_allow_feature_tier/test.py::test_allow_feature_tier_in_user 0.00s setup test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_plain_rewritable] 0.00s teardown test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings_with_old_compatibility 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_tar_zst 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_in_order 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_tar_xz 0.00s teardown test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_zip 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_native_copy_multipart 0.00s teardown test_database_delta/test.py::test_embedded_database_and_tables 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3] 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3] 0.00s setup test_backup_restore_s3/test.py::test_backup_to_tar 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 0.00s teardown test_backup_restore_s3/test.py::test_backup_with_fs_cache[False-True-False] 0.00s setup test_backup_restore_s3/test.py::test_backup_restore_system_tables_with_plain_rewritable_disk 0.00s teardown test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3_other_bucket] 0.00s setup test_dictionaries_update_and_reload/test.py::test_reload_while_loading 0.00s teardown test_dictionaries_update_and_reload/test.py::test_reload_after_fail_in_cache_dictionary 0.00s setup test_database_delta/test.py::test_embedded_database_and_tables 0.00s setup test_distributed_ddl_on_cross_replication/test.py::test_atomic_database 0.00s setup test_backup_restore_s3/test.py::test_backup_to_s3_named_collection 0.00s teardown test_attach_table_normalizer/test.py::test_attach_substr 0.00s setup test_attach_table_normalizer/test.py::test_attach_substr_restart 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_local] 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_other_bucket] 0.00s teardown test_dictionaries_update_and_reload/test.py::test_reload_after_loading 0.00s setup test_database_delta/test.py::test_multiple_schemes_tables 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_zip 0.00s teardown test_backup_restore_s3/test.py::test_backup_restore_system_tables_with_plain_rewritable_disk 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_named_collection 0.00s teardown test_backup_restore_s3/test.py::test_backup_to_s3_multipart =========================== short test summary info ============================ FAILED test_backup_restore_on_cluster/test_different_versions.py::test_different_versions ERROR test_database_delta/test.py::test_complex_table_schema - Failed: Timeou... ERROR test_database_delta/test.py::test_embedded_database_and_tables - Failed... ERROR test_database_delta/test.py::test_multiple_schemes_tables - Failed: Tim... PASSED test_allow_feature_tier/test.py::test_allow_feature_tier_in_general_settings PASSED test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_system_reload PASSED test_encrypted_disk/test.py::test_add_keys PASSED test_distributed_ddl_on_cross_replication/test.py::test_alter_ddl PASSED test_encrypted_disk/test.py::test_add_keys_with_id PASSED test_distributed_ddl_on_cross_replication/test.py::test_atomic_database PASSED test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings PASSED test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors PASSED test_distributed_ddl_on_cross_replication/test.py::test_non_query_with_table_ddl PASSED test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False] PASSED test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings_with_old_compatibility PASSED test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True] PASSED test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3] PASSED test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node0] PASSED test_disk_access_storage/test.py::test_alter PASSED test_encrypted_disk/test.py::test_backup_restore[File-local_policy-encrypted_policy-False] PASSED test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_timer PASSED test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3_other_bucket] PASSED test_allow_feature_tier/test.py::test_allow_feature_tier_in_user PASSED test_allow_feature_tier/test.py::test_it_is_possible_to_enable_experimental_settings_in_default_profile PASSED test_disk_access_storage/test.py::test_create PASSED test_attach_table_normalizer/test.py::test_attach_substr PASSED test_dictionaries_update_and_reload/test.py::test_reload_after_fail_in_cache_dictionary PASSED test_attach_table_normalizer/test.py::test_attach_substr_restart PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache] PASSED test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node1] PASSED test_disk_access_storage/test.py::test_drop PASSED test_dictionaries_update_and_reload/test.py::test_reload_after_loading PASSED test_backup_restore_s3/test.py::test_backup_restore_system_tables_with_plain_rewritable_disk PASSED test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node0] PASSED test_dictionaries_update_and_reload/test.py::test_reload_while_loading PASSED test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_local] PASSED test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3] PASSED test_create_query_constraints/test.py::test_create_query_const_constraints PASSED test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3_plain] PASSED test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[0] PASSED test_create_query_constraints/test.py::test_create_query_minmax_constraints PASSED test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_local] PASSED test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_s3_plain] PASSED test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct] PASSED test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node1] PASSED test_backup_restore_s3/test.py::test_backup_to_s3 PASSED test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[1048576] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[auto_single] PASSED test_distributed_load_balancing/test.py::test_load_balancing_default PASSED test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node0] PASSED test_disks_app_other_disk_types/test.py::test_disks_app_plain_rewritable PASSED test_dictionary_ddl_on_cluster/test.py::test_dictionary_ddl_on_cluster PASSED test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node1] PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_multipart] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_single] PASSED test_ddl_on_cluster_stop_waiting_for_offline_hosts/test.py::test_stop_waiting_for_offline_hosts PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed] PASSED test_deduplicated_attached_part_rename/test.py::test_deduplicated_attached_part_renamed_after_attach PASSED test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node0] PASSED test_enable_user_name_access_type/test.py::test_startup_scripts PASSED test_distributed_config/test.py::test_distibuted_settings PASSED test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node1] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_multipart] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_single] PASSED test_backup_restore_new/test_cancel_backup.py::test_cancel_backup PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[cache] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_multipart PASSED test_backup_restore_s3/test.py::test_backup_to_s3_named_collection PASSED test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_other_bucket] PASSED test_distributed_load_balancing/test.py::test_load_balancing_first_or_random PASSED test_backup_restore_new/test_cancel_backup.py::test_shutdown_cancel_backup PASSED test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_plain_rewritable] PASSED test_backup_restore_s3/test.py::test_backup_to_s3_native_copy_multipart PASSED test_backup_restore_s3/test.py::test_backup_to_tar PASSED test_backup_restore_s3/test.py::test_backup_to_tar_bz2 PASSED test_backup_restore_s3/test.py::test_backup_to_tar_gz PASSED test_backup_restore_s3/test.py::test_backup_to_tar_lzma PASSED test_backup_restore_s3/test.py::test_backup_to_tar_xz PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[direct] PASSED test_backup_restore_s3/test.py::test_backup_to_tar_zst PASSED test_dictionaries_dependency/test.py::test_no_lazy_load PASSED test_backup_restore_s3/test.py::test_backup_to_zip PASSED test_backup_restore_s3/test.py::test_backup_with_fs_cache[False-True-False] PASSED test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-False-False] PASSED test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-False] PASSED test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-True] PASSED test_backup_restore_s3/test.py::test_incremental_backup_append_table_def PASSED test_backup_restore_s3/test.py::test_user_specific_auth PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[flat] PASSED test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance PASSED test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[hashed] PASSED test_distributed_load_balancing/test.py::test_load_balancing_in_order PASSED test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] PASSED test_distributed_load_balancing/test.py::test_load_balancing_round_robin SKIPPED [1] test_asynchronous_metric_jemalloc_profile_active/test.py:30: Disabled for sanitizers ======== 1 failed, 95 passed, 1 skipped, 3 errors in 904.34s (0:15:04) ========= Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_t2xrta --privileged --dns-search='.' --memory=30703247360 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel0_0.jsonl --report-log-exclude-logs-on-passed-tests test_allow_feature_tier/test.py::test_allow_feature_tier_in_general_settings test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings test_allow_feature_tier/test.py::test_allow_feature_tier_in_mergetree_settings_with_old_compatibility test_allow_feature_tier/test.py::test_allow_feature_tier_in_user test_allow_feature_tier/test.py::test_it_is_possible_to_enable_experimental_settings_in_default_profile test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active test_attach_table_normalizer/test.py::test_attach_substr test_attach_table_normalizer/test.py::test_attach_substr_restart 'test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[0]' 'test_azure_blob_storage_plain_rewritable/test.py::test_insert_select[1048576]' test_backup_restore_new/test_cancel_backup.py::test_cancel_backup test_backup_restore_new/test_cancel_backup.py::test_shutdown_cancel_backup test_backup_restore_on_cluster/test_different_versions.py::test_different_versions test_backup_restore_on_cluster/test_slow_rmt.py::test_replicated_database_async 'test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3]' 'test_backup_restore_s3/test.py::test_backup_from_s3_to_s3_disk_native_copy[from_s3_to_s3_other_bucket]' test_backup_restore_s3/test.py::test_backup_restore_system_tables_with_plain_rewritable_disk 'test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_local]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_local_to_s3_plain]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_local]' 'test_backup_restore_s3/test.py::test_backup_to_disk[from_s3_to_s3_plain]' test_backup_restore_s3/test.py::test_backup_to_s3 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[auto_single]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_multipart]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[native_single]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_multipart]' 'test_backup_restore_s3/test.py::test_backup_to_s3_different_credentials[non_native_single]' test_backup_restore_s3/test.py::test_backup_to_s3_multipart test_backup_restore_s3/test.py::test_backup_to_s3_named_collection 'test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3]' 'test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_other_bucket]' 'test_backup_restore_s3/test.py::test_backup_to_s3_native_copy[policy_s3_plain_rewritable]' test_backup_restore_s3/test.py::test_backup_to_s3_native_copy_multipart test_backup_restore_s3/test.py::test_backup_to_tar test_backup_restore_s3/test.py::test_backup_to_tar_bz2 test_backup_restore_s3/test.py::test_backup_to_tar_gz test_backup_restore_s3/test.py::test_backup_to_tar_lzma test_backup_restore_s3/test.py::test_backup_to_tar_xz test_backup_restore_s3/test.py::test_backup_to_tar_zst test_backup_restore_s3/test.py::test_backup_to_zip 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[False-True-False]' 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-False-False]' 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-False]' 'test_backup_restore_s3/test.py::test_backup_with_fs_cache[True-True-True]' test_backup_restore_s3/test.py::test_incremental_backup_append_table_def test_backup_restore_s3/test.py::test_user_specific_auth test_create_query_constraints/test.py::test_create_query_const_constraints test_create_query_constraints/test.py::test_create_query_minmax_constraints test_database_delta/test.py::test_complex_table_schema test_database_delta/test.py::test_embedded_database_and_tables test_database_delta/test.py::test_multiple_schemes_tables test_ddl_on_cluster_stop_waiting_for_offline_hosts/test.py::test_stop_waiting_for_offline_hosts test_deduplicated_attached_part_rename/test.py::test_deduplicated_attached_part_renamed_after_attach 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_cache]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_direct]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_complex[complex_key_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_ranged[range_hashed]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[cache]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[direct]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[flat]' 'test_dictionaries_all_layouts_separate_sources/test_clickhouse_local.py::test_simple[hashed]' 'test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node0]' 'test_dictionaries_dependency/test.py::test_dependency_via_dictionary_database[node1]' 'test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node0]' 'test_dictionaries_dependency/test.py::test_dependency_via_explicit_table[node1]' 'test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node0]' 'test_dictionaries_dependency/test.py::test_dependency_via_implicit_table[node1]' 'test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node0]' 'test_dictionaries_dependency/test.py::test_dependent_dict_table_distr[node1]' test_dictionaries_dependency/test.py::test_no_lazy_load test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_system_reload test_dictionaries_update_and_reload/test.py::test_reload_after_fail_by_timer test_dictionaries_update_and_reload/test.py::test_reload_after_fail_in_cache_dictionary test_dictionaries_update_and_reload/test.py::test_reload_after_loading test_dictionaries_update_and_reload/test.py::test_reload_while_loading test_dictionary_ddl_on_cluster/test.py::test_dictionary_ddl_on_cluster test_disk_access_storage/test.py::test_alter test_disk_access_storage/test.py::test_create test_disk_access_storage/test.py::test_drop test_disks_app_other_disk_types/test.py::test_disks_app_plain_rewritable test_distributed_config/test.py::test_distibuted_settings test_distributed_ddl_on_cross_replication/test.py::test_alter_ddl test_distributed_ddl_on_cross_replication/test.py::test_atomic_database test_distributed_ddl_on_cross_replication/test.py::test_non_query_with_table_ddl test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority]' 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative]' test_distributed_load_balancing/test.py::test_load_balancing_round_robin test_enable_user_name_access_type/test.py::test_startup_scripts test_encrypted_disk/test.py::test_add_keys test_encrypted_disk/test.py::test_add_keys_with_id 'test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-False]' 'test_encrypted_disk/test.py::test_backup_restore[File-encrypted_policy-local_policy-True]' 'test_encrypted_disk/test.py::test_backup_restore[File-local_policy-encrypted_policy-False]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 ' returned non-zero exit status 1.